Image processing apparatus and method, and program

ABSTRACT

An image processing apparatus includes a face detection unit which detects faces from an input image, an evaluation value calculation unit which calculates an evaluation value expressing a degree of a facial expression for each of the facial expressions of the faces detected by the face detection unit, and a control unit which changes a standard for extracting an image such that an image including the facial expressions, that a small number of people have, is easily extracted, based on the number of people for each of the facial expressions of the faces detected by the face detection unit and an evaluation value for each facial expression of the face calculated by the evaluation value calculation unit.

BACKGROUND

The present disclosure relates to an image processing apparatus, animage processing method, and a program, and more particularly to animage processing apparatus, an image processing, and a program which areconfigured to obtain an image having high level of satisfaction evenwhen facial expressions of a plurality of image-captured people aredifferent.

In recent years, an image processing technology of processing a digitaloperation based on an image signal has been rapidly progressed. As theexample, a technology of detecting a face of a person from an image isdisclosed (for example, Japanese Unexamined Patent ApplicationPublication No. 2005-157679). This face detection technology has beendeveloped to be installed on a digital type imaging apparatus whichperforms imaging with a solid imaging device such as a digital stillcamera. In addition, in recent years, a technology of recognizing afacial expression of a detected face has been noticed.

For example, in Japanese Unexamined Patent Application Publication No.2004-046591, a technology, which evaluates a facial expression of a faceof an image-captured person for each captured image from an image signalin which a plurality of frames are continuously captured, and selects anappropriate image from the evaluation information, has been proposed.

Further, in Japanese Unexamined Patent Application Publication No.2009-217758, a technology which performs an evaluation as an image, notjust an evaluation of a face included in an image has been proposed.

Further, in Japanese Unexamined Patent Application Publication No.2010-117948, a technology, which has two kinds of facial expressiondetermination modes and selects a determination mode, whether the modeis a speed priority, an accuracy priority, a manual operation by a user,or the like, in response to the number of detected faces, has beenproposed.

On the other hand, in an imaging apparatus, so-called a self-timerfunction (an automatic imaging function) in which a shutter isautomatically released after a predetermined time elapses from anoperation of pressing a shutter button is generally installed not onlyin a silver-salt camera, but also in a digital still camera.

However, in the automatic imaging function, the time when the shutter isreleased is previously determined, so that it does not guarantee that animage-captured person necessarily has a good facial expression at thetime when the shutter is released and an unsatisfactory picture has beenfrequently captured.

In response, in Japanese Unexamined Patent Application Publication No.2011-030163, a technology has been proposed which conducts functionalimprovement in the automatic imaging function by changing the frequencywhen imaging and recording are performed in an automatic imagingfunction, for example.

SUMMARY

However, in the related art, there has not been a proposal relating tothe automatic imaging function in a case where the facial expressions ofa plurality of image-captured people are different.

Therefore, considering the above situation, it is desirable to obtain animage having high level of satisfaction, even when the facialexpressions of a plurality of image-captured people are different.

An image processing apparatus according to an embodiment of the presentdisclosure includes: a face detection unit which detects faces from aninput image; an evaluation value calculation unit which calculates anevaluation value expressing a degree of a facial expression for each ofthe facial expressions of the faces detected by the face detection unit;and a control unit which changes a standard for extracting an image suchthat an image including the facial expressions of a small number ofpeople are easily extracted based on the number of people for each ofthe facial expressions of the faces detected by the face detection unitand an evaluation value for each facial expression of the face iscalculated by the evaluation value calculation unit.

When there are a plurality of facial expressions that a small number ofpeople have, the control unit may change the standard such that an imageincluding a facial expression, that a largest number of children have,is easily extracted out of facial expressions that a small number ofpeople have.

When there are a plurality of facial expressions that a small number ofpeople have, the control unit may change the standard such that an imageincluding a facial expression, which people having the facialexpressions that a small number of people have are close to a center ofan angle of view, is easily extracted out of facial expressions that asmall number of people have.

The image processing apparatus may further include an extraction unitwhich extracts an image including the facial expressions that a smallnumber of people have, based on the standard changed by the controlunit.

The process by the extraction unit is a process of recording the inputimage.

The process by the extraction unit is a process of automaticallyreleasing a shutter.

The process by the extraction unit is an instruction process to a user.

The process by the extraction unit is a process of applying metadata.

The image processing apparatus may further include an imaging unit whichcaptures an object and inputs the input image, and a camera platformcontrol unit which controls an operation of a camera platform on which acasing including the imaging unit is installed.

The image processing apparatus may further include a standard adjustmentunit which adjusts the standard changed by the control unit in adirection to return the standard to an original, when an extractionprocess is performed by the extraction unit.

The standard adjustment unit may adjust the standard changed by thecontrol unit in a direction to further change the standard, when apredetermined time elapses.

The control unit may change the standard for extracting the image, bylowering or raising a threshold value corresponding to the evaluationvalue.

The control unit may change the standard for extracting the image, byapplying a gain to the evaluation value.

Further, an image processing method according to an embodiment of thepresent disclosure includes detecting faces from the input image,calculating an evaluation value expressing a degree of a facialexpression for each of facial expressions of the detected faces, andchanging a standard for extracting an image such that an image includingthe facial expressions of a small number of people, is easily extracted,based on the number of people for each of the facial expressions of thedetected faces and an evaluation value for each facial expression of thecalculated faces.

Furthermore, a program according to an embodiment of the presentdisclosure causes a computer to function as a face detection unit whichdetects faces from an input image, an evaluation value calculation unitwhich calculates an evaluation value expressing a degree of a facialexpression for each of the facial expressions of the faces detected bythe face detection unit, and a control unit which changes a standard forextracting an image such that an image including the facial expressions,that a small number of people have, is easily extracted, based on thenumber of people for each of the facial expressions of the facesdetected by the face detection unit and an evaluation value for eachfacial expression of the face calculated by the evaluation valuecalculation unit.

According to an embodiment of the present disclosure, faces are detectedfrom input image, and an evaluation value expressing a degree of afacial expression for each of facial expressions of the detected facesis calculated. Then, a standard for extracting an image is changed suchthat an image including the facial expressions, that a small number ofpeople have, is easily extracted, based on the number of people for eachof the facial expressions of the detected faces and an evaluation valuefor each facial expression of the calculated faces.

According to an embodiment of the present disclosure, an image may beextracted. Specially, an image having high level of satisfaction may beobtained, even when the facial expressions of image-captured people aredifferent.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of animaging apparatus to which the present technology is applied;

FIG. 2 is a block diagram illustrating a configuration example of acontrol unit;

FIG. 3A to 3G are diagrams illustrating an outline of the imagingprocess using an automatic imaging mode;

FIG. 4 is a flowchart illustrating an example of the imaging processusing an automatic imaging mode;

FIG. 5 is a flowchart illustrating a facial expression evaluation valuecalculation process;

FIG. 6 is a flowchart illustrating another example of the imagingprocess using an automatic imaging mode;

FIG. 7 is a flowchart illustrating still another example of the imagingprocess using an automatic imaging mode;

FIG. 8 is a flowchart illustrating an example of the imaging processusing an automatic imaging mode using a camera platform;

FIG. 9 is a flowchart illustrating an example of a threshold valueadjustment process;

FIG. 10 is a flowchart illustrating another example of the thresholdvalue adjustment process; and

FIG. 11 is a block diagram illustrating a configuration example of acomputer.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of implementing the present disclosure(hereinafter, referred to as embodiments) will be described.

Configuration Example of Imaging Apparatus

FIG. 1 is a block diagram illustrating a configuration example of animaging apparatus as an example of the imaging apparatus to which thepresent technology is applied.

An imaging apparatus 11 shown in FIG. 1 is configured by a digital stillcamera, etc., and may be installed on the camera platform 12, therebybeing connected to a camera platform 12.

The camera platform 12 may perform control using communication with theimaging apparatus 11 that is installed (connected) thereon in order tocapture an object from various angles, that is, the camera platform 12rotates the imaging apparatus 11 in the right and left directions (pan:a horizontal direction) or raises or lowers the angle of view of theimaging apparatus 11 (tilt: vertical direction). In addition, anoperation control may be configured to be embedded in the cameraplatform 12.

The imaging apparatus 11 is configured to include a lens 21, an imagingdevice 22, an analog signal processing unit 23, an A/D conversion unit24, a control unit 25, a focusing unit 26, an iris unit 27, and a zoomunit 28. Further, the imaging apparatus 11 is configured to include adisplay unit 29, a recording unit 30, an operation unit 31, a memory 32,and a camera platform corresponding communication unit 33.

The lens 21 has a focusing mechanism, an iris mechanism and a zoommechanism which are not shown. An image is formed in the imaging device22 by object light through the lens 21. The imaging device 22 isconfigured by, for example, a photoelectric conversion device such as aCharge Coupled Device (CCD) or a Complementary Metal Oxide Semiconductor(CMOS). The imaging device 22 receives the object light which is to forman image, converts the light into an electric signal, and outputs theconverted electric signal to the analog signal processing unit 23.

The analog signal processing unit 23 performs a signal process such as agamma correction or a white balance with respect to the electric signalfrom the imaging device 22, and outputs the electric signal after thesignal process to the A/D conversion unit 24. The A/D conversion unit 24performs A/D conversion with respect to the electric signal from theanalog signal processing unit 23, and outputs the digital image dataafter conversion to the control unit 25.

The control unit 25 is a control circuit which collectively controls theoperation of each unit of the imaging apparatus 11, based on theoperation signal from the operation unit 31 or the program developed inthe memory 32. That is, the control unit 25 performs a predeterminedprocess with respect to the digital image data from the A/D conversionunit 24, displays the image data after the process on the display unit29 or records the image data after the process in the recording unit 30or the memory 32. Further, the control unit 25 performs imaging controlby controlling the focusing unit 26, the iris unit 27 and the zoom unit28. Further, the control unit 25 controls the operation such as pan•tiltof the camera platform 12, in an automatic imaging mode, bycommunicating in the camera platform corresponding communication unit33.

The focusing unit 26 drives the focusing mechanism of the lens 21. Theiris unit 27 drives the iris mechanism of the lens 21. The zoom unit 28drives the zoom mechanism of the lens 21.

The display unit 29 is formed of, for example, a liquid crystal displaydevice, and displays the image corresponding to the image data fordisplay which is made in the control unit 25. The recording unit 30 isformed of an IC memory card or an embedded memory, and records the datafrom the control unit 25.

The operation unit 31 is formed of a touch panel which is layered on thedisplay unit 29, or buttons, a switch, and a dial which are provided ina casing body, and inputs a signal corresponding to the operation of auser to the control unit 25. For example, when the operation unit 31 isconfigured by the touch panel, the operation unit 31 detects anoperation of a user and inputs a signal corresponding to the detectedposition to the control unit 25.

The memory 32 is configured by a Random Access Memory (RAM) and thelike, and is used as a region in which the predetermined program isdeveloped or a temporal recording region in which data processed in thecontrol unit 25 is stored.

The camera platform corresponding communication unit 33 performscommunication with the camera platform 12 in a predeterminedcommunication manner. For example, the camera platform correspondingcommunication unit 33 transmits the operation control signal from thecontrol unit 25 to the camera platform 12, and transmits the replysignal from the camera platform 12 to the control unit 25.

Configuration Example of Control Unit

FIG. 2 is a diagram illustrating a configuration example of a controlunit in the case where an operation mode of a camera transits to anautomatic imaging mode. The transition to the automatic imaging mode isperformed in response to an instruction to cause the automatic imagingmode ON by the user through, for example, the operation unit 31.

In an example of FIG. 2, the control unit 25 is configured to include aface detection unit 51, a facial expression evaluation unit 52, athreshold value control unit 53 and an extraction unit 55.

The image data from the A/D conversion unit 24 is input to the facedetection unit 51 and the extraction unit 55. Further, the image data isinput to the display unit 29, and the still image is displayed on thedisplay unit 29.

The face detection unit 51 detects a human face from an image of imagedata from the A/D conversion unit 24, and supplies the facial expressionevaluation unit 52 with information relating to positions, sizes or thelike of the detected faces when a plurality of the detected faces arepresent. Further, in face detection, for example, the technologydisclosed in Japanese Unexamined Patent Application Publication No.2005-157679 is used. In addition, a child determination unit 61 isembedded in the face detection unit 51. The child determination unit 61determines whether there is a face of child or a face of an adult out ofthe detected human faces, supplies the facial expression evaluation unit52 with the determination result. Here, determination of whether thereis a face of child or a face of an adult is performed using eachlearning data of an adult or a child.

The facial expression evaluation unit 52 calculates an evaluation valueof a facial expression 1, an evaluation value of a facial expression 2and an evaluation value of a facial expression 3 with respect to afacial expression of each of the faces detected from the image by theface detection unit 51. Then, the facial expression evaluation unit 52determines to which one of the facial expression 1 to the facialexpression 3 a facial expression of each detected face is classified,and adds the evaluation value of the classified facial expression. Thefacial expression 1 to the facial expression 3 are the types of facialexpressions, and include, for example, a smiley face, a crying face, aproud face, an angry face and a funny face. Further, the facialexpressions may be classified into more faces, without being limited tothree kinds of facial expressions.

The facial expression evaluation unit 52 supplies the threshold valuecontrol unit 53 with the number of people having each of the facialexpression 1 to the facial expression 3 that are determined and therespective evaluation values. In addition, the facial expressionevaluation unit 52 supplies the threshold value control unit 53 withinformation relating to the position of the face detected by the facedetection unit 51 or the result of determination of whether there is achild or an adult from the child determination unit 61, as necessary.

The threshold value control unit 53 keeps respective threshold valuewith respect to each of the facial expression 1 to the facial expression3. The threshold value control unit 53 changes, for example, thethreshold value of the expression that a small number of people have,using information from the facial expression evaluation unit 52, basedon the number of people having each of the facial expression 1 to facialexpression 3 that are determined, and respective evaluation values. Thethreshold value control unit 53 supplies the extraction unit 55 with theevaluation values of the facial expression 1 to the facial expression 3and the respective threshold values corresponding to the respectiveevaluation values.

The extraction unit 55 performs a shutter process that is a process toextract (select) only one part in the image from the image data which isinput, when all of the evaluation values of the facial expression 1 tothe facial expression 3 are equal to or greater than the respectivecorresponding threshold values.

As an example of extracting an image in the extraction unit 55, theshutter process is performed with respect to the image data which isinput. Here, the shutter process represents an automatic shutterprocess. That is, the extraction unit 55 extracts the image thatsatisfies a threshold value (a standard). The extraction unit 55supplies the extracted (that is, the shutter process is performed) imagedata to the recording unit 30, and allows the extracted data to berecorded thereon.

That is, the automatic imaging mode in the imaging apparatus 11 is amode in which the automatic shutter process is performed when all theevaluation values of the facial expressions are equal to or greater thanthe respective corresponding threshold values.

In addition, other than the shutter process, examples of the imageextraction include a process which adds metadata to the correspondingimage out of a plurality of images or a process which records only thecorresponding image out of a plurality of images.

Outline of the Imaging Process Using the Automatic Imaging Mode

Next, the description about an outline of the imaging process using theabove described automatic imaging mode will be made referring to FIG. 3.

The object light forms an image in the imaging device 22 through thelens 21. The imaging device 22 receives the object light which forms animage, converts the light into an electric signal, and outputs theconverted electric signal to the analog signal processing unit 23. Theanalog signal processing unit 23 performs signal processes such as agamma correction or a white balance with respect to the electric signalfrom the imaging device 22, and outputs the electric signal after thesignal process to the A/D conversion unit 24. The A/D conversion unit 24performs A/D conversion with respect to the electric signal from theanalog signal processing unit 23, and outputs the digital image dataafter conversion to the control unit 25.

The control unit 25 outputs the digital image data to the display unit29, and the image 101 corresponding to the digital image data which isshown in FIG. 3A is displayed on the display unit 29. The image 101, inwhich three people are on the upper side and a person having a cryingfacial expression is on the lower middle side, is displayed on thedisplay unit 29. The people on the upper side have no facial expression,a smiley facial expression and a smiley facial expression from the left.

In this status, the user sets the automatic imaging mode to “ON” usingthe operation unit 31. The control unit 25 reads the polling process ofthe automatic imaging mode ON which is input through the operation unit31 and changes the operation mode of the imaging apparatus 11 into theautomatic imaging mode. Then, the control unit 25 overlaps the image 102of “the automatic imaging mode ON” on the image 101 of the display unit29, as shown in FIG. 3B, and activates respective parts in FIG. 2.

The face detection unit 51 detects four human faces from the image ofimage data from the A/D conversion unit 24, and supplies the facialexpression evaluation unit 52 with information such as the positions andthe sizes of the detected faces.

First, as illustrated by dots in FIG. 3C, the facial expressionevaluation unit 52 calculates the evaluation value of each facialexpression with respect to the person on the upper left side. The facialexpression of the person on the upper left side is determined as nofacial expression, based on the evaluation value of each detected facialexpression. In a case of no facial expression, the facial expression ofthe person is not classified into any one. In addition, after transitionto the automatic imaging mode, the image 103 expressing “the automaticimaging mode ON” is displayed on the lower left side in a smaller sizethan the image when the transition to the automatic imaging mode is onthe process.

Next, the facial expression evaluation unit 52 calculates the evaluationvalue of each facial expression, as shown with dots in FIG. 3D, withrespect to the person on the upper middle side. The facial expression ofthe person on the upper middle side is classified as a smiley face (afacial expression 1), based on the evaluation value of each calculatedfacial expression, and the number of the smiley face becomes two.

Next, the facial expression evaluation unit 52 calculates the evaluationvalue of each facial expression, as shown with dots in FIG. 3E, withrespect to the person on the upper right side. The facial expression ofthe person on the upper right side is classified as a smiley face (afacial expression 1), based on the evaluation value of each calculatedfacial expression, and the number of the smiley face becomes three.

Next, the facial expression evaluation unit 52 calculates the evaluationvalue of each facial expression, as shown with dots in FIG. 3F, withrespect to the person on the lower middle side. The facial expression ofthe person on the lower middle side is classified as a crying face (afacial expression 2), based on the evaluation value of each calculatedfacial expression, and the number of the crying face becomes one.Further, the dots in FIGS. 3C to 3F are shown only for the convenienceof explanation, and are not shown in the real image 101.

When the facial expression evaluation unit 52 compares the number ofpeople whose facial expression are detected and the number four ofpeople within the angle of view with the number four of people detectedwithin the angle of view, if all numbers are same, the facial expressionevaluation unit 52 ends the facial expression evaluation calculationprocess. The threshold value control unit 53 changes the standardreferring to the evaluation value and the number of each facialexpression in the image 101. For example, the threshold value controlunit 53 changes the standard by lowering the threshold value of thecrying face that the smallest number of people have, by one step. Thethreshold value control unit 53 supplies the extraction unit 55 with theevaluation values of the facial expression 1 to the facial expression 3and the corresponding threshold values.

Then, the extraction unit 55 performs the shutter process when each ofthe evaluation values of the facial expression 1 to the facialexpression 3 exceeds the respective corresponding threshold value andthe evaluation values of the facial expression 2 (crying face) exceedsthe threshold value of which standard is changed. The extraction unit 55performs the shutter process which is a process for extracting the stillimage from the image data which is input as shown in FIG. 3G, andsupplies the recording unit 30 with the still image data on which theshutter process is performed, that is, extracted, and allows theextracted data to be recorded thereon.

That is, in the automatic imaging mode, when each of the evaluationvalues of all facial expressions exceeds the respect threshold value,the still image is extracted. Accordingly, even if all the people do nothave smiley faces, that is, facial expressions are different, the stillimage is extracted.

Then, in the change of the standard, by lowering the threshold value byone step, the determination standard of the determination process withrespect to the image extraction is loosened. The determination standardis loosened, so that the facial expression of which standard is changed,that is, the facial expression that the smallest number of people haveis easily extracted.

There are many cases, in which the person, having a face different fromthe faces of the surrounding people, in other words, the person having aface of the facial expression that the smallest number of people have,may be a central character or a leading character, preferably, the imageof that person has been desired to be extracted (taken). Thus, byperforming as above, the image of the person (a leading role) having thefacial expression that the smallest number of people have may becaptured more.

Further, although the threshold value is lowered by one step, the numberof the step is not limited to one. In addition, a standard changingmethod is not limited to the lower the threshold value, if it is astandard changing method in which an image including the expression thatthe small number of people have is easily extracted, any method may beused. For example, the threshold value of the facial expression that alarge number of people have may be raised, or a gain more than one maybe applied to the evaluation value of the facial expression that a smallnumber of people have.

Example of Imaging Process Using the Automatic Imaging Mode

Next, the imaging process using the automatic imaging mode of theimaging apparatus 11 will be described referring to FIG. 4. For example,in the process described above, the user sets the automatic imaging modeto ON using the operation unit 31. In response, the operation mode ofthe imaging apparatus 11 is transited to the automatic imaging mode bythe control unit 25, thereby being started.

The imaging device 22 receives the object light which forms an imagethrough the lens 21, converts the light into an electric signal, andoutputs the converted electric signal to the analog signal processingunit 23. The analog signal processing unit 23 performs a signal processsuch as a gamma correction or a white balance with respect to theelectric signal from the imaging device 22, and outputs the electricsignal after the signal process to the A/D conversion unit 24. Then, theimage data that is A/D converted by the A/D conversion unit 24 is inputto the face detection unit 51.

In step S11, the face detection unit 51 detects human faces from theimage of the image data from the A/D conversion unit 24, and calculatesthe number of people within the angle of view based on the detectedhuman faces. In step S12, the face detection unit 51 determines whetherthe number of people within the angle of view is equal to or greaterthan one.

In step S12, when the face detection unit 51 determines that the numberof people within the angle of view is equal to or greater than one, theface detection unit 51 supplies the facial expression evaluation unit 52with the information relating to positions, sizes or the like of thedetected faces, and then the process proceeds to step S13.

In step S13, the facial expression evaluation unit 52 performs thefacial expression evaluation calculation process. The detaileddescription of the facial expression evaluation calculation process willbe made below, referring to FIG. 5. Through the process of step S13, thedetermined number of people of each of the facial expression 1 to thefacial expression 3 and respective evaluation values are calculated. Thecalculated number of people of each of the facial expression 1 to thefacial expression 3 and respective evaluation values are supplied to thethreshold value control unit 53.

In step S14, the threshold value control unit 53 determines whetherthere is a facial expression that the smallest number of people have,referring to the number of people of each of the facial expression 1 tothe facial expression 3 and respective evaluation values from the facialexpression evaluation unit 52. For example, when the number of thefacial expression 1 (a smiley face) is two, the number of the facialexpression 2 (a crying face) is one and the number of the facialexpression 3 (a proud face) is two, in step S14, it is determined thatthere is the facial expression that the smallest number of people have,and then the process proceeds to step S15.

In step S15, the threshold value control unit 53 lowers the thresholdvalue of the facial expression 2 (a crying face) that is a facialexpression that the smallest number of people have, by one step.Further, in an example of FIG. 4, even in a case when there is aplurality of facial expressions that the smallest number of people have,the process proceeds to step S15. In step S15, the threshold value islowered with respect to the facial expressions that the smallest numberof people have. Then, the threshold value control unit 53 supplies theextraction unit 55 with the evaluation values of the facial expression 1to the facial expression 3 and the respective threshold valuescorresponding to the respective evaluation values.

In step S14, in a case where there is no facial expression that thesmallest number of people have, the step S15 is skipped, and then theprocess proceeds to step S16. Further, even when the number of peoplewithin the angle of view is either one or two, in step S14, it isdetermined that there is no facial expression that the smallest numberof people have. Then, the threshold value control unit 53 supplies theextraction unit 55 with the evaluation values of the facial expression 1to the facial expression 3 and the respective threshold valuescorresponding to the respective evaluation values.

In step S16, the extraction unit 55 determines whether the evaluationvalue of each facial expression respectively exceeds the correspondingthreshold value. In step S16, in a case where it is determined that theevaluation value of each facial expression respectively exceeds thecorresponding threshold value, the process proceeds to step S17.

In step S17, the extraction unit 55 extracts the image from the imagedata which is input. The extraction unit 55 supplies the recording unit30 with the extracted (that is, the shutter process is performed) imagedata, and allows the extracted data to be recorded thereon. Further, thenumber of times that the shutter process is performed is not limited toone, but the shutter process may be performed a plurality of times.

In step S16, when it is determined that the evaluation value of eachfacial expression does not respectively exceed the correspondingthreshold value, that is, when it is determined that there is at leastone of expression of which evaluation value does not exceed thecorresponding threshold value, the step S17 is skipped and then theprocess proceeds to step S18.

Alternatively, in step S12, when the face detection unit 51 determinesthat the number of people within the angle of view is zero, the stepsS13 to S17 are skipped, and then the process proceeds to step S18.

In step S18, the face detection unit 51 determines whether there is aprocess end instruction from the user through the operation unit 31. Instep S18, when it is determined that there is not the process endinstruction, the process returns to step S11, and then the subsequentprocesses are repeated.

In step S18, when there is the process end instruction, the processusing the automatic imaging mode is ended.

As described above, when it is determined that there is a facialexpression that the smallest number of people have, the threshold valueof the facial expression that the smallest number of people have islowered, and thereby increasing the possibility of obtaining the imageincluding the facial expression that the smallest number of people have(that is, the expression of the person who may be in the center).

Further, in the case of the automatic imaging mode, in the process ofstep S18 it is determined whether the image is extracted. If the imageis extracted once, the imaging process of FIG. 4 may be ended.

In addition, when the imaging process of FIG. 4 is repeated, like theprocess of the automatic imaging mode in a case of using the cameraplatform 12 which will be described referring FIG. 8, after extractingthe image, the threshold value adjustment may be performed.

Facial Expression Evaluation Value Calculation Process

The facial expression evaluation value calculation process of step S13in FIG. 4 will be described referring to the flow chart in FIG. 5.

The facial expression evaluation unit 52 sets N=1 in step S31 andcalculates the facial expression evaluation value of the first person instep S32. For example, with regard to the facial expression of the firstperson, the evaluation value of the facial expression 1 expressing thedegree of the facial expression 1, the evaluation value of the facialexpression 2 expressing the degree of the facial expression 2 and theevaluation value of the facial expression 3 expressing the degree of thefacial expression 3 are respectively calculated.

In step S33, the facial expression evaluation unit 52 classifies thefacial expression of the first person into any one facial expression ofthe facial expression 1 to the facial expression 3, based on thecalculated evaluation value of each facial expression. In the facialexpression evaluation unit 52, the classification is performed, forexample, using the learned data with respect to each facial expression.In addition, the threshold value is used even at the time ofclassification in step S33, but the threshold value is smaller than thethreshold value that is used to determine the time when an imageextraction is performed, in step S16 in FIG. 4.

The facial expression evaluation unit 52 adds one to the number ofpeople of the classified facial expression in step S34, and adds theevaluation value of the facial expression of the first person that isclassified, out of the evaluation values calculated in the step S32, tothe evaluation value of the classified facial expression in step S35.

The facial expression evaluation unit 52 sets N=N+1 in step S36, anddetermines whether N is no more than the number of people within theangle of view in step S37. In step S37, when it is determined that N isno more than the number of people within the angle of view, the processreturns to step S32, and then the subsequent processes are repeated. Instep S37, when it is determined that N is larger than the number ofpeople within the angle of view, the facial expression evaluation valuecalculation process is ended.

Another Example of the Imaging Process Using the Automatic Imaging Mode

Next, an example of children priority in the imaging process using theautomatic imaging mode of the imaging apparatus 11 will be describedreferring to a flowchart in FIG. 6. In addition, the processes of stepsS51 to S54 and steps S59 to S61 in FIG. 6 fundamentally are the same asthe processes of steps S11 to S14 and steps S16 to S18 in FIG. 4 and thedetailed description is repeated so that the description will beomitted.

In step S51, the face detection unit 51 detects a human face from animage of image data from the A/D conversion unit 24, and calculates thenumber of people within the angle of view, based on the detected humanface. At this time, the child determination unit 61 determines whetherthere is a face of child or a face of an adult out of the detected humanfaces. In step S52, the face detection unit 51 determines whether thenumber of people within the angle of view is one or more.

When the face detection unit 51 determines that the number of peoplewithin the angle of view is one or more in step S52, the face detectionunit 51 supplies the facial expression evaluation unit 52 withinformation such as the positions and the sizes of the detected faces,and then the process proceeds to step S53. The determination result ofwhether there is a child or an adult from the child determination unit61 is also supplied to the facial expression evaluation unit 52 asnecessary.

In step S53, the facial expression evaluation unit 52 performs theaforementioned facial expression evaluation value calculation process.Through the process of the step S53, the number of people of each of thefacial expression 1 to the facial expression 3 which are determined andthe respective evaluation values are calculated. The calculated numberof people of each of the facial expression 1 to the facial expression 3and the respective evaluation values are supplied to the threshold valuecontrol unit 53.

In step S54, the threshold value control unit 53 determines whetherthere is a facial expression that the smallest number of people have,referring to the number of people of each of the facial expression 1 tothe facial expression 3 and the respective evaluation values from thefacial expression evaluation unit 52. In step S54, when it is determinedthat there is a facial expression that the smallest number of peoplehave, the process proceeds to step S55.

In step S55, the threshold value control unit 53 determines whetherthere are a plurality of facial expressions that the smallest number ofpeople have. In step S55, when it is determined that there is only onefacial expression that the smallest number of people have, the processproceeds to step S56. In step S56, the threshold value control unit 53lowers the threshold value of the facial expression determined as thatthe smallest number of people have by one step, and supplies theextraction unit 55 with the evaluation values of the facial expression 1to the facial expression 3 and the respective threshold valuescorresponding to the respective evaluation values, and then the proceedsto step S59.

For example, when the number of the facial expression 1 (a smiley face)is three, the number of the facial expression 2 (a crying face) is twoand the number of the facial expression 3 (a proud face) is two, thereare a plurality of facial expressions that the smallest number of peoplehave, so that in step S55, it is determined that there are a pluralityof facial expressions that the smallest number of people have, and thenthe process proceeds to step S57.

For example, even in a case of having the same number of facialexpressions such as a case where the number of the facial expression 1is one, the number of the facial expression 2 is one and the number ofthe facial expression 3 is one, and a case where the number of thefacial expression 1 is two, the number of the facial expression 2 is twoand the number of the facial expression 3 is two, or in another casewhere the number of the facial expression 1 is one, the number of thefacial expression 2 is one and the number of the facial expression 3 istwo, in step S55, it is determined that there are a plurality of facialexpressions that the smallest number of people have.

Here, the determination resulting from the child determination unit 61as to whether there is a face of child or a face of an adult is suppliedto the threshold value control unit 53 through the facial expressionevaluation unit 52. In step S57, the threshold value control unit 53determines whether there is a facial expression that a large number ofchildren have out of a plurality of facial expressions that the smallestnumber of people have. That is, when the facial expression of one personout of two people having the facial expression 2 (a crying face) is thefacial expressions of children and the facial expressions of two out oftwo people having the facial expression 3 (a proud face) are the facialexpressions of children, the facial expression 3 is determined as thefacial expression that a large number of children have, out of thefacial expressions that the smallest number of people have.

In step S57, when it is determined that there is a facial expressionthat a large number of children have out of a plurality of facialexpressions that the smallest number of people have, it proceeds to stepS58. In step S58, the threshold value control unit 53 lowers thethreshold value of the facial expression (the facial expression 3) thata large number of children have, out of a plurality of facialexpressions that the smallest number of people have by one step, andsupplies the extraction unit 55 with the evaluation values of the facialexpression 1 to the facial expression 3 and the respective thresholdvalues corresponding to the respective evaluation values, and then theprocess proceeds to step S59.

In step S57, when it is determined that there is not a facial expressionthat a large number of children have, out of a plurality of facialexpressions that the smallest number of people have, the process of thestep S58 is skipped, and then the process proceeds to step S59. At thistime, the threshold value control unit 53 supplies the extraction unit55 with the evaluation values of the facial expression 1 to the facialexpression 3 and the respective threshold values corresponding to therespective evaluation values.

In step S59, the extraction unit 55 determines whether the evaluationvalue of each of the facial expressions respectively exceeds thecorresponding threshold value. In step S59, when it is determined thatthe evaluation value of each of the facial expressions respectivelyexceeds the corresponding threshold value, the process proceeds to stepS60. In step S60, the extraction unit 55 extracts an image from an inputimage data. The extraction unit 55 supplies the recording unit 30 withthe extracted (that is, the shutter process is performed) image data,and allows the extracted data to be recorded thereon.

Further, in step S59, when it is determined that there is at least onefacial expression of which the evaluation value does not exceed thecorresponding threshold value, the process of the steps S60 is skipped,the process proceeds to step S61.

Alternatively, in step S52, when the face detection unit 51 determinesthat the number of people within the angle of view is zero, the stepsS53 to S60 are skipped, and then the process proceeds to step S61.

In step S61, the face detection unit 51 determines whether there is aprocess end instruction from the user through the operation unit 31. Instep S61, when it is determined that there is not the process endinstruction, the process returns to step S52, and then the subsequentprocesses are repeated.

In step S61, when it is determined that there is a process endinstruction, the imaging process using the automatic imaging mode isended.

As described above, when it is determined that there are a plurality offacial expressions that the smallest number of people have, it isconfigured to lower the threshold value of the facial expression that alarge number of children have, out of a plurality of facial expressionsthat the smallest number of people have, and thereby increasing thepossibility of obtaining the image that is the facial expression thatthe smallest number of people have and includes the facial expressionthat a large number of children have in that expression.

In a group in which a child is present, an image is frequently capturedwith the child as a center, and thereby increasing the possibility ofobtaining the image of various facial expressions of a child.

Another Example of an Imaging Process Using the Automatic Imaging Mode

Next, an example of the center priority of the angle of view in theimaging process using the automatic imaging mode of the imagingapparatus 11 will be described referring to a flowchart in FIG. 7. Inaddition, the processes of steps S81 to S84 and steps S89 to S91 in FIG.7 fundamentally are the same as the processes of steps S11 to S14 andsteps S16 to S18 in FIG. 4 and the detailed description is repeated sothat the description will be omitted.

In step S81, the face detection unit 51 detects human faces from theimage of the image data from the A/D conversion unit 24, and calculatesthe number of people within the angle of view, based on the detectedhuman faces. In step S82, the face detection unit 51 determines whetherthe number of people within the angle of view is equal to or greaterthan one.

In step S82, when the face detection unit 51 determines that the numberof people within the angle of view is equal to or greater than one, theface detection unit 51 supplies the facial expression evaluation unit 52with information relating to positions, sizes or the like of thedetected faces, and then the process proceeds to step S83.

In step S83, the facial expression evaluation unit 52 performs thefacial expression evaluation calculation process which is describedabove referring to FIG. 5. Through the process of step S83, the numberof people of each of the facial expression 1 to the facial expression 3which are determined and respective evaluation values are calculated.The calculated number of people of each of the facial expression 1 tothe facial expression 3 and respective evaluation values are supplied tothe threshold value control unit 53.

In step S84, the threshold value control unit 53 determines whetherthere is a facial expression that the smallest number of people have,referring to the number of people of each of the facial expression 1 tothe facial expression 3 and the respective evaluation values from thefacial expression evaluation unit 52. In step S84, when it is determinedthat there is a facial expression that the smallest number of peoplehave, the process proceeds to step S85.

In step S85, the threshold value control unit 53 determines whetherthere are a plurality of facial expressions that the smallest number ofpeople have. In step S85, when it is determined that there is only onefacial expression that the smallest number of people have, the processproceeds to step S86. In step S86, the threshold value control unit 53lowers the threshold value of the facial expression determined as thatthe smallest number of people have by one step, and supplies theextraction unit 55 with the evaluation values of the facial expression 1to the facial expression 3 and the respective threshold valuescorresponding to the respective evaluation values, and then the processproceeds to step S89.

For example, when the number of the facial expression 1 (a smiley face)is three, the number of the facial expression 2 (a crying face) is twoand the number of the facial expression 3 (a proud face) is two, it isdetermined that there are a plurality of facial expressions that thesmallest number of people have, and then the process proceeds to stepS87.

Here, the information of the position of the face detected by the facedetection unit 51 is supplied to the threshold value control unit 53through the facial expression evaluation unit 52. In step S87, thethreshold value control unit 53 determines whether there is a facialexpression of a person who is closest to the center of the angle of viewout of the facial expressions that the smallest number of people have.For example, when the person closest to the center of the angle of viewhas a crying face, it is determined that there is the facial expression(the facial expression 2: a crying face) of the person closest to thecenter of the angle of view, out of the facial expressions that thesmallest number of people have, and then the process proceeds to stepS88.

In step S88, the threshold value control unit 53 lowers the thresholdvalue of the facial expression (the facial expression 2: a crying face)of the person closest to the center of the angle of view out of thefacial expressions that the smallest number of people have by one step,supplies the extraction unit 55 with the evaluation values of the facialexpression 1 to the facial expression 3 and the respective thresholdvalues corresponding to the respective evaluation values, and then theprocess proceeds to step S89.

In step S87, when it is determined that there is no a facial expressionof a person who is closest to the center of the angle of view out of thefacial expressions that the smallest number of people have, the step S88is skipped, and then the process proceeds to step S89. At this time, thethreshold value control unit 53 supplies the extraction unit 55 with theevaluation values of the facial expression 1 to the facial expression 3and the respective threshold values corresponding to the respectiveevaluation values.

In step S89, the extraction unit 55 determines whether the evaluationvalue of each facial expression respectively exceeds the correspondingthreshold value. In step S89, when it is determined that the evaluationvalues of each facial expression respectively exceeds the correspondingthreshold value, the process proceeds to step S90. In step S90, theextraction unit 55 extracts the image from the image data which isinput. The extraction unit 55 supplies the recording unit 30 with theextracted (that is, the shutter process is performed) image data, andallows the extracted data to be recorded thereon.

Further, in step S89, when it is determined that there is at least oneexpression of which the evaluation values does not exceed thecorresponding threshold value, the step S90 is skipped, and then theprocess proceeds to step S91.

Alternatively, in step S82, when the face detection unit 51 determinesthat the number of people within the angle of view is zero, the stepsS83 to S90 are skipped, and then the process proceeds to step S91.

In step S91, the face detection unit 51 determines whether there is aprocess end instruction from the user through the operation unit 31. Instep S91, when there is not the process end instruction, the processreturns to step S82, and subsequent processes are repeated.

In step S91, when it is determined that there is the process endinstruction, the imaging process using automatic imaging mode is ended.

As described above, when it is determined that there are a plurality offacial expressions that the smallest number of people have, it isconfigured to lower the threshold value of the same facial expression asthe facial expression of the person closest to the center of the angleof view, out of a plurality of facial expressions. Accordingly, theabove fact increases the possibility of obtaining the image includingthe facial expression which the smallest number of people have and inwhich the facial expression of the person who is in the center of theangle of view is included.

For example, a central character may be mostly located in the center ofthe angle of view, like a bride and a bridegroom being located in thecenter of the angle of view in a wedding ceremony. Accordingly, theabove fact increases the possibility of obtaining the image includingvarious facial expressions of the central character.

In addition, the example of children priority in FIG. 6 and the exampleof the center priority of the angle of view in FIG. 7 may be performedin parallel. In this case, for example, in the case of “No” in step S57in FIG. 6, steps S87 and S88 in FIG. 7 may be inserted. In contrast, inthe case of “No” in step S87 in FIG. 7, steps S57 and S58 in FIG. 6 maybe inserted.

In addition, the present technology is applicable to a function ofautomatically releasing the shutter in many times using the cameraplatform 12. This function is described in Japanese Unexamined PatentApplication Publication No. 2011-030163. An example of a case where thepresent technology is applied in this function will be described belowas an automatic imaging mode using the camera platform 12.

Example of an Imaging Process Using an Automatic Imaging Mode Using theCamera Platform

The imaging apparatus 11 sets an operation mode of a camera to theautomatic imaging mode, using the camera platform 12. Accordingly, inthe imaging apparatus 11, that uses the camera platform 12 as a resultof rotating the imaging apparatus 11 in the right and left direction(pan: a horizontal direction) or raising or lowering the angle of viewof the imaging apparatus 11 (tilt: vertical direction), various objectsin the vicinity of the imaging apparatus 11 are captured multiple times.

That is, in the imaging process, in a case of an automatic imaging modeusing the camera platform 12, the shutter process (an image extraction)is frequently repeated until there is the process end instruction. Thus,in this case, the threshold value control unit 53 operates as athreshold value adjustment unit which performs an adjustment process toadjust the standard (threshold value) that is changed prior to return tothe original value, each time when determining whether the evaluationvalue of the facial expression exceeds the threshold value.

Next, an example of the imaging process in the automatic imaging modeusing the camera platform 12 of the imaging apparatus 11 will bedescribed referring to a flowchart in FIG. 8. In addition, the stepsS111 to S117 in FIG. 8 fundamentally are the same as the steps S11 toS17 in FIG. 4, and the detailed description is repeated so that thedescription will be omitted.

After it is determined that the evaluation value of each facialexpression exceeds the respective threshold value in step S116 and theimage is extracted in step S117, the process proceeds to step S118.Alternatively, after it is determined that at least one evaluation valueof each facial expression does not exceed the corresponding thresholdvalue in step S116, the process proceeds to step S118.

In step S118, the threshold value control unit 53 performs a thresholdvalue adjustment process. An example of the threshold value adjustmentprocess will be described referring to a flowchart in FIG. 9.

In step S131, the threshold value control unit 53 determines whether theimage is extracted in step S117. In step S131, when it is determinedthat the image is extracted, the process proceeds to step S132.

In step S132, the threshold value control unit 53 raises the thresholdvalue of the facial expression. In addition, the facial expression as anobject of which the threshold value is adjusted may be the facialexpression of which the threshold value is lowered in step S115, thefacial expression that a small number of people have, or the facialexpression of which the evaluation value is highest, compared to thepredetermined threshold value. That is, the facial expression of anobject of which the threshold value is raised is not limited to theabove facial expressions, but various methods are applicable.

In addition, the standard adjustment method is not limited to a methodof raising the threshold value which is similar to the standard changemethod, but, for example, a method of lowering the threshold value or anadjustment of a gain with respect to the evaluation value is acceptable.

On the other hand, in step S131, when it is determined that the image isnot extracted in step S117, step S132 is skipped, the threshold valueadjustment process is ended, and then the process proceeds to step S119.

In step S119, the face detection unit 51 determines whether there is aprocess end instruction from the user through the operation unit 31. Instep S119, when there is not a process end instruction, the imagingprocess returns to step S111, and the subsequent processes are repeated.Further, in a case of the automatic imaging mode using the cameraplatform 12, prior to this step S111, the control unit 25 controls theoperations of the focusing unit 26, the iris unit 27, the zoom unit 28and the camera platform 12 connected through the camera platformcorresponding communication unit 33, and changes the range for imaging.

In step S119, when it is determined that there is the process endinstruction, the process using the automatic imaging mode is ended.

As described above, in using the camera platform 12, it is configured toadjust the standard, every time when the image is extracted, so that thethreshold value may be prevented from being excessively lowered.Further, capturing the same composition may be suppressed. Accordingly,without being limited to only the composition thereof, for example, thesmiley face, various compositions may be captured.

In addition, in the example of FIG. 9, the example of the process in theautomatic imaging mode that is described above referring to FIG. 4 isexplained as an example of a case of using the camera platform 12, butthe process is not limited to the example. That is, even in a case ofusing the camera platform 12, the example of children priority describedreferring to FIG. 6 and the example of the center priority of the angleof view described referring to FIG. 7 may be performed.

Example of the Threshold Value Adjustment Process

Further, referring to a flowchart in FIG. 10, another example of thethreshold value adjustment process of step S118 in FIG. 8 will bedescribed. In addition, the steps S151 and S152 in FIG. 10 fundamentallyare the same as the steps S131 to S132 in FIG. 9 so that the detaileddescription will be omitted.

In step S151, the threshold value control unit 53 determines whether theimage is extracted in step S117 of FIG. 8. In step S151, when it isdetermined that the image is extracted, the process proceeds to stepS152.

In step S152, the threshold value control unit 53 raises the thresholdvalue of the facial expression, and then the process proceeds to stepS153.

On the other hand, in step S151, when it is determined that the image isnot extracted in step S117, the step S152 is skipped, and then theprocess proceeds to step S153.

In step S153, the threshold value control unit 53 determines whether thenumber of an embedded timer is N (a predetermined time) or more. Inaddition, the timer is started in step S156 described later.

In step S153, when it is determined that the number of the timer is N ormore, that is, when it is determined that time elapses by N, the processproceeds to step S154. In step S154, the threshold value control unit 53lowers the threshold value of the facial expression that is raised instep 152, and in step S155, the timer is set to zero.

In step S153, when it is determined that the number of the timer is lessthan N, that is, when it is determined that time does not yet elapse N,the steps S154 and S155 are skipped, and then the process proceeds tostep S156.

In step S156, the threshold value control unit 53 starts the timer instep S156, and thus ends the threshold value adjustment process.

As described above, the threshold value which has been raised by thethreshold value adjustment every time when the image is extracted, islowered every predetermined time. Accordingly, for example, since thereis a concern that the threshold value is excessively increased soon dueto the fact that the threshold value has been raised by the thresholdvalue adjustment every time when the image is extracted, it is possibleto prevent the image from not being extracted.

Further, in an example of FIG. 10, if a predetermined time elapses, thethreshold value is lowered it is possible to configure that the numberof times that the image is not extracted is counted, and if the timeexceeds a predetermined number, the threshold value is lowered.

Furthermore, in the above explanation, as an example of an imageextraction process, the example of performing a shutter process isexplained, but the image extraction process is not limited to theshutter process. The image extraction in the present technology may beapplied to an image extraction of a recorded image that is performed,for example, at the time of an image process of a recorded image, at thetime of reproduction of a recorded image, or the like. Further, as anexample, the present technology may be applied to a case where adetermination of an image extraction is previously performed and storedas metadata, and then the image extraction is processed later.

In addition, as an example of an image extraction, it is described thata shutter is automatically released, but it is possible to use aninstruction process to a user that indicates (instructs) to a user thatnow is a shutter chance, without releasing the shutter.

Further, as an instruction process to a user, the present technology maybe applied to an indication (an audio output) to prompt a user toperform a certain process (in this case, an image extraction process)such as “This image may be extracted”.

Furthermore, the present technology may be applied to a digital videocamera recorder, without being limited to a still camera. In a case ofbeing applied to the digital video camera recorder, for example, it ispossible to record metadata together with a moving picture, extract astill picture from a moving picture, or apply metadata. The applicationof metadata corresponds to an extraction. Accordingly, it is possible torecognize an important image when reproducing an image. In addition, ina case of a digital video camera recorder, it is possible to extract onewhich meets the present condition (including application of metadata)while recording a moving picture.

The present technology may be applied to a case where a compositiondetermination is not processed in a camera without a recording function,that is, a camera side, but is performed in a receiving side (areceiving side (a server) of a personal computer or a monitoring camerasystem).

Further, the present technology may be applied to a printer, an album, athumbnail or a slide show.

That is, specifically, as a printer, it is considered to print out theimage which is extracted based on the present technology from aplurality of images which are previously recorded. As an album or aslide show, it is considered to make an album or a slide show from aplurality of extracted images. In addition, an album is made of an imagewhich is extracted based on the present technology. In addition, a viewof a thumb nail is made of an extracted image, with an image extractedbased on the present technology as a thumb nail.

The aforementioned series of processes may be implemented using hardwareor software. In a case of implementing the aforementioned series ofprocesses using software, programs which construct the software areinstalled from a program recording media into a computer incorporatedinto a dedicated hardware, a general purpose computer in which variousprograms are installed in order to perform various functions, or thelike.

Configuration Example of a Computer

FIG. 11 is a block diagram illustrating a configuration example ofhardware of a computer to execute a series of processes in a program.

In a computer, a Central Processing Unit (CPU) 201, a Read Only Memory(ROM) 202, and a Random Access Memory (RAM) 203 are connected to eachother through a bus 204.

In addition, an input/output interface 205 is connected to the bus 204.An input unit 206, an output unit 207, a recording unit 208, acommunication unit 209 and a drive 210 are connected to the input/outputinterface 205.

The input unit 206 includes a keyboard, a mouse, a microphone, and thelike. The output unit 207 includes a display, a speaker and the like.The recording unit 208 includes a hard disk, a non-volatile memory, andthe like. The communication unit 209 includes a network interface andthe like. The drive 210 drives the removable media 211 such as amagnetic disk, an optical disk, a magneto-optical disk or asemiconductor memory.

In the computer configured like the above, the CPU 201 loads the programstored in the recording unit 208 to the RAM 203 through the input/outputinterface 205 and the bus 204 and performs the program, and thus theaforementioned series of processes are performed.

The program which is executed by the computer (CPU 201) may be recordedin the removable media 211 as, for example, a package media to beprovided. In addition, a program may be provided through a wire orwireless transmission media such as a Local Area Network, Internet, ordigital broadcasting.

In a computer, by mounting the removable media 211 in the drive 210, aprogram may be installed in the recording unit 208 through theinput/output interface input/output interface 205. Further, a programmay be received in the communication unit 209 through a wire or wirelesstransmission media and installed in the recording unit 208. In addition,a program may previously be installed in the ROM 202 or the recordingunit 208.

Further, the program that the computer executes may be a program ofwhich process is performed in time series along the order explained inthe present disclosure or a program of which the process is performed inparallel or at the necessary timing such as when a call is received.

Further, in the present specification, the term of system means anoverall apparatus configured by a plurality of devices, blocks, means,and the like.

Further, the embodiment in the present disclosure is not limited to theabove embodiment, and various modifications may be made withoutdeparting from the spirit of the present disclosure.

Although the preferred embodiment of the present disclosure has beendescribed above referring to the accompanying drawings, the disclosureis not limited the embodiment. It should be understood by those skilledin the art that various modifications, combinations, sub-combinationsand alterations may occur depending on design requirements and otherfactors insofar as they are within the scope of the appended claims orthe equivalents thereof.

Further, the present technology may have the following configurations.

(1) An image processing apparatus including a face detection unit whichdetects faces from an input image, an evaluation value calculation unitwhich calculates an evaluation value expressing a degree of a facialexpression for each of the facial expressions of the faces detected bythe face detection unit, and a control unit which changes a standard forextracting an image such that an image including the facial expressions,that a small number of people have, is easily extracted, based on thenumber of people for each of the facial expressions of the facesdetected by the face detection unit and an evaluation value for eachfacial expression of the face calculated by the evaluation valuecalculation unit.

(2) In the image processing apparatus according to (1), when there are aplurality of facial expressions that a small number of people have, thecontrol unit changes the standard such that an image including a facialexpression, that a largest number of children have, is easily extractedout of facial expressions that a small number of people have.

(3) In the image processing apparatus according to (1) or (2), whenthere are a plurality of facial expressions that a small number ofpeople have, the control unit changes the standard such that an imageincluding a facial expression, which people having the facialexpressions that a small number of people have are close to a center ofan angle of view, is easily extracted out of facial expressions that asmall number of people have.

(4) The image processing apparatus according to any one of (1) to (3)further includes an extraction unit which extracts an image includingthe facial expressions that a small number of people have, based on thestandard changed by the control unit.

(5) In the image processing apparatus according to (4), the process bythe extraction unit is a process of recording the input image.

(6) In the image processing apparatus according to (4), the process bythe extraction unit is a process of automatically releasing a shutter.

(7) In the image processing apparatus according to (4), the process bythe extraction unit is an instruction process to a user.

(8) In the image processing apparatus according to (4), the process bythe extraction unit is a process of applying metadata.

(9) The image processing apparatus according to (4) further includes animaging unit which captures an object and inputs the input image, and acamera platform control unit which controls an operation of a cameraplatform on which a casing including the imaging unit is installed.

(10) The image processing apparatus according to (9) further includes astandard adjustment unit which adjusts the standard changed by thecontrol unit in a direction to return the standard to an original, whenan extraction process is performed by the extraction unit.

(11) In the image processing apparatus according to (10), the standardadjustment unit adjusts the standard changed by the control unit in adirection to further change the standard, when a predetermined timeelapses.

(12) In the image processing apparatus according to any one of (1) to(11), the control unit changes the standard for extracting the image, bylowering or raising a threshold value corresponding to the evaluationvalue.

(13) In the image processing apparatus according to any one of (1) to(11), the control unit changes the standard for extracting the image, byapplying a gain to the evaluation value.

(14) An image processing method for an image processing apparatusincludes detecting faces from input image, calculating an evaluationvalue expressing a degree of a facial expression for each of facialexpressions of the detected faces, and changing a standard forextracting an image such that an image including the facial expressions,that a small number of people have, is easily extracted, based on thenumber of people for each of the facial expressions of the detectedfaces and an evaluation value for each facial expression of thecalculated faces.

(15) A program for causing a computer to function as a face detectionunit which detects faces from an input image, an evaluation valuecalculation unit which calculates an evaluation value expressing adegree of a facial expression for each of the facial expressions of thefaces detected by the face detection unit, and a control unit whichchanges a standard for extracting an image such that an image includingthe facial expressions, that a small number of people have, is easilyextracted, based on the number of people for each of the facialexpressions of the faces detected by the face detection unit and anevaluation value for each facial expression of the face calculated bythe evaluation value calculation unit.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2012-048721 filed in theJapan Patent Office on Mar. 6, 2012, the entire contents of which arehereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An image processing apparatus comprising: a facedetection unit which detects faces from an input image; an evaluationvalue calculation unit which calculates an evaluation value expressing adegree of a facial expression for each of the facial expressions of thefaces detected by the face detection unit; and a control unit whichchanges a standard for extracting an image such that an image includingthe facial expressions, that a small number of people have, is easilyextracted, based on the number of people for each of the facialexpressions of the faces detected by the face detection unit and anevaluation value for each facial expression of the face calculated bythe evaluation value calculation unit.
 2. The image processing apparatusaccording to claim 1, wherein when there are a plurality of facialexpressions that a small number of people have, the control unit changesthe standard such that an image including a facial expression, that alargest number of children have, is easily extracted out of facialexpressions that a small number of people have.
 3. The image processingapparatus according to claim 1, wherein when there are a plurality offacial expressions that a small number of people have, the control unitchanges the standard such that an image including a facial expression,which people having the facial expressions that a small number of peoplehave are close to a center of an angle of view, is easily extracted outof facial expressions that a small number of people have.
 4. The imageprocessing apparatus according to claim 1, further comprising: anextraction unit which extracts an image including the facial expressionsthat a small number of people have, based on the standard changed by thecontrol unit.
 5. The image processing apparatus according to claim 4,wherein the process by the extraction unit is a process of recording theinput image.
 6. The image processing apparatus according to claim 4,wherein the process by the extraction unit is a process of automaticallyreleasing a shutter.
 7. The image processing apparatus according toclaim 4, wherein the process by the extraction unit is an instructionprocess to a user.
 8. The image processing apparatus according to claim4, wherein the process by the extraction unit is a process of applyingmetadata.
 9. The image processing apparatus according to claim 4,further comprising: an imaging unit which captures an object and inputsthe input image; and a camera platform control unit which controls anoperation of a camera platform on which a casing including the imagingunit is installed.
 10. The image processing apparatus according to claim9, further comprising: a standard adjustment unit which adjusts thestandard changed by the control unit in a direction to return thestandard to an original, when an extraction process is performed by theextraction unit.
 11. The image processing apparatus according to claim10, wherein the standard adjustment unit adjusts the standard changed bythe control unit in a direction to further change the standard, when apredetermined time elapses.
 12. The image processing apparatus accordingto claim 1, wherein the control unit changes the standard for extractingthe image, by lowering or raising a threshold value corresponding to theevaluation value.
 13. The image processing apparatus according to claim1, wherein the control unit changes the standard for extracting theimage, by applying a gain to the evaluation value.
 14. An imageprocessing method for an image processing apparatus, comprising:detecting faces from input image; calculating an evaluation valueexpressing a degree of a facial expression for each of facialexpressions of the detected faces; and changing a standard forextracting an image such that an image including the facial expressions,that a small number of people have, is easily extracted, based on thenumber of people for each of the facial expressions of the detectedfaces and an evaluation value for each facial expression of thecalculated faces.
 15. A program for causing a computer to function as aface detection unit which detects faces from an input image; anevaluation value calculation unit which calculates an evaluation valueexpressing a degree of a facial expression for each of the facialexpressions of the faces detected by the face detection unit; and acontrol unit which changes a standard for extracting an image such thatan image including the facial expressions of a small number of people,is easily extracted, based on the number of people for each of thefacial expressions of the faces detected by the face detection unit andan evaluation value for each facial expression of the face calculated bythe evaluation value calculation unit.